Skip to content

[Feature] Add scalar quantization support for vector extension#28

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache
Jul 18, 2026
Merged

[Feature] Add scalar quantization support for vector extension#28
adsharma merged 1 commit into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache

Conversation

@rahul-iyer

@rahul-iyer rahul-iyer commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds scalar quantization support for HNSW vector indexes behind the quantization option.

This PR supports:

  • quantization := 'sq8'
  • quantization := 'sq16'
  • quantized in-memory distance computation for ANN traversal
  • optional full-precision reranking via use_full_precision_rerank
  • SIMD-friendly SQ16 dot-product kernel with scalar fallback
  • quantized query-vector/cache alignment fixes
  • visibility checks for quantized on-disk embeddings
  • quantized cache invalidation on index mutation

Dot product with quantization is rejected for now because the quantized metric path currently supports
L2/L2SQ/Cosine-style scoring.

Behavior

Without quantization, HNSW continues to use the existing full-precision float embedding path.

With quantization enabled:

  • index traversal uses quantized embeddings
  • query vectors are quantized once per query
  • cached quantized payloads store compact integer vectors plus scale/norm metadata
  • if use_full_precision_rerank := true, final candidates are rescored using full-precision embeddings
  • if use_full_precision_rerank := false, results are returned directly from the quantized metric path

Benchmark

Ran it on OpenAI 50k small

Results:

Variant Load Index Build Ingest Query Latency DB Size Bytes/Vec Query RSS Recall@10 Precision@10
base 5.862s 51.000s 56.862s 17.812 ms/query 606.5 MiB 12718.7 746.6 MiB 0.9960 0.9960
sq8 5.765s 13.111s 18.875s 16.366 ms/query 723.4 MiB 15170.2 327.1 MiB 0.9460 0.9460
sq16 5.421s 23.337s 28.758s 16.444 ms/query 908.0 MiB 19041.5 489.4 MiB 0.9950 0.9950

Speedup vs baseline:

Variant Load Index Build Ingest Query Latency Query RSS Reduction
sq8 1.02x 3.89x 3.01x 1.09x 2.28x lower
sq16 1.08x 2.19x 1.98x 1.08x 1.53x lower

Storage overhead vs baseline:

Variant DB Size Bytes/Vec
sq8 1.19x 1.19x
sq16 1.50x 1.50x

Correctness Notes

This PR also fixes several issues found during testing:

  • quantized query buffer sizing/alignment
  • SQ8/SQ16 cached metadata layout
  • traversal using the quantized metric function instead of the full-precision metric
  • stale quantized cache after index mutation
  • visibility checks for quantized on-disk embeddings

Quantized cache invalidation is currently coarse-grained: mutations invalidate the full quantized cache.
A future improvement should make this transaction-aware and row/version based.

@rahul-iyer rahul-iyer changed the title [DRAFT][Feature] Add scalar quantization support for vector extension [Feature] Add scalar quantization support for vector extension Jul 9, 2026
@adsharma
adsharma self-requested a review July 13, 2026 19:44

@adsharma adsharma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Couple of minor comments.

Tests should be green once the main repo's commit is in.

Comment thread vector/src/index/hnsw_index.cpp
Comment thread vector/src/index/hnsw_graph.cpp
Persist SQ8 and SQ16 embeddings in an internal _QEMB node table so
quantized index state participates in Ladybug's normal MVCC, WAL, and
checkpoint lifecycle.

Use the durable quantized rows for HNSW construction and on-disk search,
with batched table-backed reads when caching is disabled and an aligned
dense cache when caching is enabled. Preserve optional full-precision
reranking and normalize cosine vectors for the quantized distance path.

Keep the shared dense cache transactionally safe by tracking dirty writer
transactions and invalidating it from commit and rollback callbacks. Add
coverage for quantized index behavior and document storage layout, query
flow, cache semantics, and benchmark results.
@rahul-iyer
rahul-iyer force-pushed the quantized-rerank-cache branch from c0b7d47 to 907df8f Compare July 17, 2026 05:10
@adsharma
adsharma merged commit 07d82ae into LadybugDB:main Jul 18, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants